GENERALITY AND OBJECTIVITY Central Issues in Putting a Dialogue Evaluation Tool into Practical Use

This paper presents a first set of test results on the generality and objectivity of the Dialogue Evaluation Tool DET. Building on the assumption that most, if not all, dialogue design errors can be viewed as problems of non-cooperative system behaviour, DET has two closely related aspects to its use. Firstly, it may be used for the diagnostic evaluation of spoken human-machine dialogue. Following the detection of miscommunication, DET enables in-depth classification of miscommunication problems that are caused by flawed dialogue design and supports the repair of those problems, preventing their future occurrence. Secondly, DET can be used to guide early dialogue design in order to prevent dialogue design errors from occurring in the implemented system. We describe the development and in-house testing of the tool, and present the results of ongoing work on testing its generality and objectivity on an external corpus, i.e. an early corpus from the Sundial project in spoken language dialogue systems development.